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DETAILED ACTION 
Response to Amendment 
1. The amendment filed on 06/26/2003 has been entered. Claims 1, 2, 5-7, 1 1-14, 16-18, 20 
and 22 have been amended. Claims 3, 4, 9, 10, 15 and 19have been canceled. 



Claim Rejections - 35 USC §103 

2. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

3. Claims 1-2, 5-8, 1 1-14, 16-18, 20-22 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Xiong et al. U.S. Patent No. 6,434,265 (hereinafter Xiong), and further in view 
of Shum et al. U.S. Patent No. 6,271,855 (hereinafter Shum). 

4. Claim 1: 

(1) Xiong teaches an image synthesis method comprising: 

A placement information obtaining step of obtaining placement information about a 
plurality of images in which adjacent images have a common subject region (e.g., in column 4, 
lines 5-40, Xiong teaches a method for constructing a panorama from rectilinear images in 3D 
through projective registration and calibration including: (1) the projective registrations of 
overlapping images, (2) calibration and global optimization of these images, a self calibration in 
which 2D image planes are positioned as 3D planes in space); 
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A setting step of setting one mapping mode out of a plurality of mapping modes each 
corresponding to a different mapping surface in accordance with said obtained placement 
information (e.g., in column 8, lines 18-58 of Xiong, it is stated that "overlapping photographs 
are analyzed to determine what orientation the photographs were taken in order to establish a 
common ground for subsequent operations and the panorama is constructed on a particular 
geometry that will best facilitate the rendering of the projection of the panorama onto a chosen 
viewing plane for viewing". The Xiong discloses some typical geometry on which panoramas are 
formed; In column 8, lines 18-58, Xiong further discloses that panorama is constructed on a 
particular geometry that will best facilitate the rendering of the projection onto a chosen viewing 
plane such as cubic, polyhedral, cylindrical and spherical geometries); and 

A synthesis step of combining said plurality of images by using the mapping mode set in 
said setting step (e.g., in column 4, lines 5-40 of Xiong, it is stated that the composing or 
blending in which images are ready to be re-projected to a 3D environment map with pixels in 
overlap regions being composed from multiple; In column 8, lines 18-58, Xiong further teaches 
that overlapping photographs are analyzed to determine what orientation the photographs were 
taken in order to establish a common ground for subsequent operations and the panorama is 
constructed on a particular geometry that will best facilitate the rendering of the projection of the 
panorama onto a chosen viewing plane for viewing. The Xiong further discloses some typical 
geometry on which panoramas are formed); 

A changing step, of changing the mapping mode (In column 4, lines 40-50, it is stated 
that "the projection module may be controlled through the user interface 230 as well, to allow a 
user to select what geometry will be projected onto". Therefore, Xiong teaches a changing step 
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through the user interface 230 by selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface). 

- The Examiner interprets "a placement information obtaining step of obtaining placement 
information about a plurality of images in which adjacent images have a common subject 
region" as an automatic registration and calibration step of registering the overlapping 
images and capturing common overlapping areas between overlapping images and 
minimizing the average squared pixel intensity difference with respect to certain 
transformation parameters. 

- The Examiner interprets the mapping mode as mapping images onto a geometric surface 
such as a planar or a cylindrical surface (Applicant's specification, page 1, lines 20-25). 
Accordingly, the Examiner interprets the setting step of setting a mapping mode as the 
selecting step of selecting a geometric surface. In column 8, lines 18-58, Xiong discloses 
that panorama is constructed on a particular geometry that will best facilitate the 
rendering of the projection onto a chosen viewing plane (a geometric surface) such as 
cubic, polyhedral, cylindrical and spherical geometries. In column 4, lines 40-50, it is 
stated that "the projection module may be controlled through the user interface 230 as 
well, to allow a user to select what geometry will be projected onto". Therefore, Xiong 
teaches a selecting step of selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface in accordance with the 
obtained placement information. As applied to the present application, Xiong fulfills the 
claimed limitation of a setting step of setting one mapping mode out of a plurality of 
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mapping modes each corresponding to a different mapping surface in accordance with 
said obtained placement information. 

(2) However, it is not clear whether Xiong implicitly teaches the claimed limitation of "a 
generating step, of issuing , when an image formed by changing the mapping mode in said 
changing step does not comply with a predetermined condition set in accordance with the 
mapping mode, a warning and generating a synthesized image in accordance with the 
predetermined condition" in its totality (i.e., generation of a warning message). 

(3) Shum implicitly teaches the claimed limitation of a generating step, of issuing , when 
an image formed by changing the mapping mode in said changing step does not comply with a 
predetermined condition set in accordance with the mapping mode, a warning and generating a 
synthesized image in accordance with the predetermined condition (Shum abstract; column 18, 
lines 55-65). 

(4) It would have been obvious to one of ordinary skill in the art to have incorporated the 
Shum's warning message generation step because Xiong suggests a generating step of generating 
a synthesized image in accordance with the predetermined condition (Xiong column 3, lines 35- 
55; column 17, lines 15-67; column 18, lines 1-4). Moreover, Xiong teaches in figures 2 and 3 a 
user interface and a global optimization that provides feedback to the computer system such as 
issuing warning messages on the computer monitor 218 when the pair-wise objective function is 
not desirable for a poor selection of the projection viewing plane (and the resulting panoramas 
will have imperfectly aligned images that give shadow or ghosting effects) due to a poor 
selection of the projection viewing plane and the resulting panoramas will have imperfectly 
aligned images that give shadow or ghosting effects. Xiong also points to a seamless multi- 
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resolution average blending method that would result in an absent of shadow effects (column 14, 
lines 1-45). Xiong further teaches a human interaction being present throughout the image 
synthesis process to provide feedback to the computer system in all the nonlinear optimizations 
to let users monitor the progress of the system and allow them to intervene when necessary and 
therefore the claimed limitation suggests an obvious modification of Xiong because providing 
feedback to users is similar to issuing a message to users in a user interaction with the nonlinear 
optimizations when the changing step does not comply with a predetermined condition set or 
user-selected parameter set. 

(5) One having the ordinary skill in the art would have been motivated to do this because 
this would have provided the user the informative message such as issuing a warning message 
through the user interaction when convergence to the predetermined condition setting cannot be 
obtained in the image synthesis (Xiong column 17, lines 15-67, column 18, lines 1-4) for non- 
solvable optimization problems (Shum column 18, lines 38-67). 



Claim 2 recites all the limitations of claim 1 and adds the limitation of "a focal length 
obtaining step." Xiong teaches finding projective parameters such as 3D rotation parameters 
(pan, tilt roll), center of projection of images, ratio of focal lengths, and the like (column 10, 
lines 9-28). Xiong implicitly teaches finding the camera internal parameters vector of which the 
focal length is a component (column 11, lines 15-42). 



Claim 2: 



Claim 5: 
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The claim 5 recites all the limitations of claim 1 and adds the limitation of "a displaying 
step of displaying a cuttable rectangular region " Xiong teaches how to align images more 
precisely by changing the coordinates for positioning an image. Xiong further teaches placing the 
images 1210 at selected tangents on the viewing sphere 1220 (figure 12, and column 17, lines 
12-65). 

5. Claim 6: 

The claim 6 encompasses the same scope of invention as that of claim 1 except additional 
claimed limitation of "an image synthesis apparatus" and "a displaying step of displaying a 
cuttable rectangular region". However, Xiong further discloses in figure 3 an apparatus for image 
synthesis. Xiong teaches how to align images more precisely by changing the coordinates for 
positioning an image. Xiong further teaches placing the images 1210 at selected tangents on the 
viewing sphere 1220 (figure 12, and column 17, lines 12-65). 

Claim 7: 

The claim 7 recites all the limitations of claim 6 and adds the limitation of "a focal 
length obtaining step." The Xiong reference teaches finding projective parameters such as 3D 
rotation parameters (pan, tilt roll), center of projection of images, ratio of focal lengths, and the 
like (column 10, lines 9-28). Xiong implicitly teaches finding the camera internal parameters 
vector in his image synthesis apparatus of figure 3 because the focal length is a component of 
that vector (column 11, lines 15-42). 

Claim 8: 
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The claim 8 recites all the limitations of claim 6 and adds the limitation of "a changing 
step of changing the mapping mode." Xiong teaches that panorama is constructed on a particular 
geometry that will best facilitate the rendering of the projection of the panorama onto a chosen 
viewing plane such as cubic, polyhedral, cylindrical and spherical geometries (column 8, lines 
18-58). 



6. Claim 11: 

The claim 1 1 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of a computer-readable storage medium having a program for 
implementing image synthesis method. However, Xiong further discloses the claimed limitation 
of a computer-readable storage medium having a program for implementing image synthesis 
method (e.g., in column 3, lines 54-57, it is stated "a program residing in system memory 220 
which stores output data and other data"). 

Claim 12: 

The claim 12 recites all the limitations of claim 1 1 and adds u a focal length obtaining 
step." Xiong teaches finding projective parameters such as 3D rotation parameters (pan, tilt roll), 
center of projection of images, ratio of focal lengths, and the like (column 10, lines 9-28). Xiong 
implicitly teaches finding the camera internal parameters vector of which the focal length is a 
component (column 11, lines 15-42). 

Claim 13: 

The claim 13 recites all the limitations of claim 1 1 and adds the limitation of "a changing 
step of changing the mapping mode." Xiong teaches that panorama is constructed on a particular 
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geometry that will best facilitate the rendering of the projection of the panorama onto a chosen 
viewing plane such as one of the cubic, polyhedral, cylindrical and spherical geometries (column 
8, lines 18-58). 

7. Claim 14: 

The claim 14 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of an image synthesis method comprising a generating step of 
generating coordinate-space transformation parameters. However, Xiong et al. further discloses 
the claimed limitation of a generating step of generating coordinate-space transformation 
parameters (for example, in a local pair-wise registration, the estimation of parameters in a 
projective matrix which represents a coordinate transformation matrix; column 1 1, lines 1-46). 

Claim 16: 

The claim 16 recites an image synthesis method comprising a reference position-setting 
step. The Xiong reference teaches a multi-resolution weighted average blending for perspective 
alignment (column 14, lines 2-45). Xiong teaches for example in column 1 1 setting or estimating 
camera orientation and the rotation matrix that is based on camera orientation. 

Claim 17: 

The claim 17 recites all the limitations of claim 14 and adds the limitation of "a storage 
step of storing a generated panoramic synthesized image." Xiong teaches panoramic canvas 
which can be simply be a buffer or memory in a computer system such as memory 220 of figure 
2 and onto which the blended images is copied (column 15, lines 15-28). 
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8. Claim 18: 

The claim 18 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of an image synthesis apparatus comprising a generating means for 
generating coordinate-space transformation parameters and image synthesis means for 
combining the images. However, Xiong further discloses the claimed limitation of an image 
synthesis apparatus comprising a generating means for generating coordinate-space 
transformation parameters and image synthesis means for combining the images (e.g., a pair- 
wise registration module 222 and the projection function module 228. The parameters in a 
projective matrix are estimated which represents an image coordinate transformation matrix. The 
projection function module 228 constructs a panoramic scene by projecting the blended image 
onto any designated geometry view surface; column 4, lines 21-52, and column 1 1, lines 1-46). 

Claim 20: 

The claim 20 recites all the limitations of claim 18 and adds the limitation of "a reference 
position setting means " Xiong teaches for example in column 1 1 setting or estimating camera 
orientation on which the rotation matrix is based. 

Claim 21: 

The claim 21 recites all the limitations of claim 18 and adds the limitation of "storage 
means for storing a generated panoramic synthesized image, coordinate transformation 
parameters and coordinate-space transformation parameters." Xiong teaches storage means such 
as panoramic canvas which can be simply a buffer or memory in a computer system such as 
memory 220 in figure 2 and onto which the blended images is copied (column 15, lines 15-28). 
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Xiong teaches that the improved projective matrix parameters (i.e., the coordinate and 
coordinate-space transformation parameters) can be used to generate improved estimation of 
camera parameters and blending may be further iterated after an initial blending (column 13, 
lines 53-67). Finally, Xiong teaches projective parameters, e.g., 3D rotation parameters, center of 
projection of images, ratio of focal lengths, and an affine transformation in constructing a two- 
dimensional view of the environment (column 10, lines 16-28). 
9. Claim 22: 

The claim 22 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of a computer-readable storage medium having a program for 
implementing image synthesis method and a generating step of generating coordinate-space 
transformation parameters. However, Xiong further discloses the claimed limitation of a 
computer-readable storage medium having a program for implementing image synthesis method 
and a generating step of generating coordinate-space transformation parameters (e.g., a program 
residing in system memory 220 which stores output data and other data; column 3, lines 54-67. 
Xiong also teaches in a local pair-wise registration the estimation of parameters in a projective 
matrix which represents a coordinate transformation matrix; column 11, lines 1-46. Finally, the 
Xiong reference teaches that the improved projective matrix parameters will in turn be used to 
generate improved estimation of camera parameters and blending may be further iterated after an 
initial blending; column 13, lines 53-67). 



Conclusion 



# 
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10. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

a. Szeliski et al. U.S. Patent No. 6,097,854 discloses an image mosaic construction 
system and apparatus with patch-based alignment. 

b. Kumar et al. U.S. Patent No. 5,963,664 discloses a system for generating three- 
dimensional mosaics from a plurality of input images. 

1 1 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (703) 605-1213. 
The examiner can normally be reached on 8:00 AM - 4:30 PM 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (703) 305-4713. The fax phone numbers for the 
organization where this application or proceeding is assigned are (703) 308-6606 for regular 
communications and (703) 308-6606 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is (703) 395-3900. 
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